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ABSTRACT 

The actual impact of research and demonstration 
projects in vocational education is evaluated. Criteria for a 
reliable research design are specified regarding sample size, sample 
selection, control groups, and economic and educational benefit and 
cost evaluation. The impact of research and demonstration projects is 
analyzed in the areas of curriculum^ teacher education, learning 
processes, capital equipment and facilities, supportive services, and 
innovative input combinations. The actual contribution of these 
projects to vocational education is examined in two aspects: The 
chciract eristics of research and demonstration projects that satisfy 
the ideal criteria in actual implementation are discussed, and the 
extent to which educators and policy makers have made use of the 
research and demonstration projects is evaluated. The general 
conclusion is that although some demonstration projects have helped 
modify or improve instruction techniques, limitations still exist: 
(1) Lack of adequate research design or evaluation methodology, (2) 
abundance of descriptive studies, (3) weakness in survey technique, 
and (4) weakness of impact due to small sample size, limited 
population, or triviality of problem tested. A table displays a 
sampling of a variety of projects funded under Parts C, D, and I of 
the vocational education amendments. (NJ) 
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I , Introduction 

Since the landmark Vocational Education Act of 
total expenditures on vocational educati.on research and 
demonstration (R&D) projects have been estimated at about 
$275 million, with more than 2500 projects supported by 
the. Federal Government. A logical question for policy-- 
mcikcrsi and funding agencies to ask is v/hc^t is the impact 
or contribution made by these K&D projects to vocational 
education? This is an especially pertinent question 
since Vac Vocational Education Act,- including its R&D 
provision, is scheduled to be examined by Congress during 
this year. Tlius, the actual contribution made by these 
U&D projects must be assessed so that this information 
may be useful for policymakers. 

I^is paper v;ill first specify some important criteria 
for a R&D design that can be used in actual implementation 
and generalization. These criteria include the required 
sample size relative to the size of expected impact, the 
randomness of the sample selection, the control ox, com- 
parison groups, and the economic and educational benefit/ 
cost evaluation .of these projects. 
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The major task of this paper is to evaluate the 
actual 'impact of these RSlD projects. In order to evaluate 
their actual contributions to vocational education, it is 
necessary to identify two types of impact on vocational 
education: (1) the intermedi ate impact on vocational 
cducatnon such as the modification or revision of curri- 
cula, the reallocation of funds within the educational 
system (either at the local. State, or Federal level), 
the effects on students' aptitudes and school performance, 
the number of graduates produced, and the percent, of 
graduates V70rl:ing in occupations for which they were pre- 
pared; and, (2) the final output of these programs, such 
as the labor market performances of these students in 
terms of v/ages, employment, and earnings or increased 
job satisfaction. Some of these impact evaluative ques- 
tions are unansv/erable given the nature of published 
materials and the limited resources of this assigned study, 
Nevertheless, this paper will provide a tentative assess- 
ment based on available R&D results and will also suggest 
a methodology for future analysis of the impact of applied 
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R&D projects In vocational education which should yield 
much more concrete results in terms of unambiguous esti 
mates of net educational and economic impact. 
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II. General Criteria for [-ieasuring the Educational 



AND Economic Impact of RsD Projects 

It is coinnionplacc judgment that the major problems 
in the evaluation of social and educational programs are 
that the goals of these progrcjms are vague, though the 
positive promises of such programs are strongly asserted, 
while the estimated end results of these programs demon- 
strate weak effects. The work concerning R&D projects in 
vocational education is no exception to this situation. 
Therefore, in order for meaningful application of the 
claimed results of any R&D program, several criteria 
• should be fulfilled. These are: 

1. Specification of the Objectives ; One of 

the major tasks in developing a R&D project 
in vocational education is to specify the 
objectives of the project which are consis- 
tent with the objectives of vocational educa- 
tion overall. For instance, the objectives 
of R&D projects vary in terms of the point 
at which the researcher focuses on the process 
of the R&D project such as (a) enrollment 
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advanced or new and emerging occupations; 

(b) enrollment in advanced technical educa- 
tion programs, according to Section 108 of 
P.L. 90-576 in the U.S. Congress; or 

(c) preparation of individuals for gainful 
employment. These stated oljjectives can be 
summarized in terms of three tangible meas- 
urements: employnvLrrt gain, income change, 
and educational attainment. These objectives 
can be called "ultimate goals." There are 
other types of goals which can be called 
"intermediate goals" such as the improvement 
in students* scores or attitude in the voca- 
tional education, the improvement in student 
attendance in vocat ional education , or a 
senr^e of fulfillm.ent of the vocational teachers 
after having developed a new program. It 
should be noted, however, that the achieve- 
ment of certain "intermediate goals" is not 
necessarily consistent with the ultimate 
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goals. Furthermore, there may not be uni- 
form agreement on the relative importance 
of these goals, but it is important that 
researchers and adjnin ir^t ra tors recognize 
thes.e alternative objectives and tlie neces- 
sity for assigning relative v;eights to them. 
Such a system of prioritized goals then 
allows one to focus the research effort on 
the most important outputs of the research 
or demonstration project. 

2. AT:? propr iate E xpori mental Design : For a R&D 
proj ect to be useful for actual implementa- 
tion, the project should have an appropriate 
experimental design. This appropriateness 
implies the random assignment of a target 
population to an experimental and a control 
group, and a sample size which will allow 
one to detect statistically significant ef- 
fects should these exist, \fhen natural 
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experiments are in question, there can be 
no random assignment, whereupon the choice 
of one or more comparison groups becomes 
critical. If the control or comparison 
group is not properly selected, the con- 
clusions or the effectiveness of this study- 
becomes uncertain. Therefore, regardless 
of the true facts of the case it becomes 
difficult if not impossible to detect 
whether such projects have any actual im- 
pact on odvicatxonal performance. 

The size of sample is also an important 
consideration in designing a study. If a 
study is carried in one classroom or v;ith 
small number of students, the ability to 
infer from results of this small to a large 
universe becomes limited. For instance, it 
is very difficult to generalize the results 
of the demonstration project of 50 students 
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in an art education study by Lockette (1966) 
to other student populations in the country • 

Another important consideration is the random- 
ness of sample selection. Random assignment 
allov;s one to make inferences of causality 
V7ith a degree of certainty v;hich is simply 
not feasible when one uses other research de- 
signs. In addition, for a given target popu- 
lation in order to replicate the- results of 
a R&D project/ the sample part icipants should 
be randomly drawn from that population so that 
they arc representative to the total popula- 
tion rather than a special group of partici- 
pants, A serious source of bias in research 
design in most social and educational R&D 
projects continues to be self -select ion bias. 
The fact that a participant is interested in 
participation in a project is perhaps due to 
the fact that his motivation, background, and 
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• interests are different from non-participants. 
Without controlling or adjusting for this 
type of bias, the conclusions from any R&D 
project again may not be valid in actual 
appl icat ion . 

3 • Prope r Tools of Research and Evaluation : \^hen 
R&D projects are carried out, the basic con- 
corn ::£; to GyCamine the effects of program 
inputs on program outputs. In social and 
educational programs, especially those with 
multiple objectives and, hence, multiple out- 
putsj, there are no unambiguous measurements 
of the net effects of an R&D project. An 
output is a result of the interaction of many 
socio-dcmosgraphic/ educational and economic 
forces. A descriptive comparison of one or 
more gross output performances does not 
properly control for these interaction forces. 
A commonly used technique for measuring the 

1 1 
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net impact of a program input is the use of 
multiple regression analysis. Each estimated 
regression coefficient becomes the net impatt 
of an input variable on the output variable, 
holding the effects of other factors constant. 
Althov\gh this procedure is an .improvement over 
gross descriptive comparison, it does not • 
necessarily control for all other variables. 
In addition, even when these effects can be 
measured, the questions of costs of the R&D 

i 

inputs relative to the benefit; of the R&D 
outputs have to be considered.: If there are 
positive educcitional or economic effects but 
the relevant educational or economic- costs 
are higher than benefits, a program should 
not be implemented on economic efficiency 
grounds. However, it should be noted that 
a finding of "no positive effect" or benefits 
less than costs does not necessary mean a 
negative judgment for a R&D as an informational 
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exercise. The fact of no significant effect 
or even negative effect of a program is a con- 
tribution to knowledge so that po3 Icymakers 
do not waste resources on the program in the 
future and alternative program can thus be 
considered. The important point is that* 
whether a program is effect ive or not , the 
method of evalviation should be carried out 
properly / so the result of implementat ion is 
an accurate one. 

I 

In conclusion/ only if RS<D projects satisfy the 
above criteria will the measured contribution to improve 
vocational cducati.on be reliable. The next section 
of this paper examines the actual contr ibv^tion of R&D 
in vocational education in tvN^o aspects. One aspect is 
to discuss the character is tics and aspects of these RS<D 
projects that satisfy the ideal criteria in actual im- 
plementation. The other aspect is to evaluate the ex- . 
tent to which educators and policymakers have made use 
of these research and demonstration projects. 

13 
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III, Evaluation of the RsD Projects 
And Their Actual Impact 

The purpose of this paper is not to review all the 
research and demonstration projects that have been carried 
out in vocational education. Given time and resource 
constraints, such a task would be impossible. Rather, 
this paper will examine the actual impact and contributions 
of R&D studies with respect to specific inputs into the 
learning process. For a comprehensive understanding of 
those R<ScD studies, the ERIC series on review and synthesis, 
of vocational education can serve as a basic source. 
Readers may consu],t with these studies such as those by 
Lanham and Trytten (1966), Larson (19G6), Phipps and 
VJarmbrod (1956) # Schaffer and Tuckman (1956), Phillips 
and Briggs (1969), h^allace (1970), Stromsdorfer (1972) 
and Miller and Miller (1974). 

This i^oper will not review all these R&D studies, 
but it will discuss the quality and general findings of 
the R&D projects reported therein to the extent that 
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these projects make meaningful contributions to advancing 
the art and science of vocational education. 

Two major types of research projects in vocational 
education can be discerned: one is research on teaching 
and learning techniques in vocational education. This 
type of research cons iders the type of content which 
vocational students should learn and how to teach- this 
content most effect i.vcly . Such types are development 
oriented. In most cases, evaJ.uations of tliese pilot 
demonstration projects were made so that project investi- 
gators could decide whether their implementat i.on should 
be continued. The other type of research is designed 
to study tho. results or outcomes of conventional voca-- 
tional education. This type of research generally 
analyzes eitlier the economic costs or benefits of conven- 
tional vocational education separately or the joint com- 
parison of these economic costs and benefits. The- first 
of these tv/o types of studies is the concern of this 
paper. 

15 
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Research and Dononntrat ion Projects 

Among the many hundreds of research and 
demonstration projects in vocational education, most 
are small scale and 1 imited in scope to their local 
environment. Very few demonstration projects provide 
dramatic evidence about tlic value of their instructional 
techniques or content as opposed to extant or conven- 
tional content or technique. Even though some of these 
v;ere I'eported as successful experiments, their sample 
sizes v/ere too small to report statistically reliable 
result s« And, their case study nature makes their 
extension of their reported results to the population 
at ]arge questionable. V7hat makes the situation worse 
is that many of these experiments were not properly 
designed and thus provided dubious results. Regrettably, 
this is a situation v;hich literally occurs time and 
time again. 

It is best to organ izc these studies in terms of 
the way in wliich they attempted to change student be- 
havior and capabilities via changing the types and mix 
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of educational inputs. These educational inputs can be 
grouped in the following broad categories: 

1. curriculum; 

2 . teacher education , teaching methods , or 
teaching teachers to teach; 

3. learning processes, of which a subset is 
instruct ional materials and devices ; 

4. capital equipment and facilities; 

5. supportive services, which include guidance, 
counseling, testing, ai:id labor market infor- 
mation; and 

6 • innovative input coivb inations, which include 
career educat ion . 



17 



Curriculum , 

Curriculum development concerns the specifica- 
tron of the information or knowledge components 
needed to change behavior or performance in a desired 
way. Such questions are asked as "vrnat knowledge and 
skills does one need to become a skilled machinist, 
airline stewardess, or* vocational educator?" Legions 
of curriculuju guides have been devised. Yet few have 
been tested in. an experimental setting. Factor 
analys is of s3:ill comi^onont s in non-exper imental 
settings has been used to identify various critical 
competencic^s , but the incremental impact of each of 
these competencies or skill inputs on behavior 
in almost every case has not been measured. Taxonomies 
are presented based on expert judgment and the general 
operational procedure from this point is to extend 
the curricula to as many locales as possible. 
Extensive effort is spent on "merchandising" the 
curriculum packages even though conclusive evidence 
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of the directio n of impact on performance, much 
less the quantitative impact, is usually lacking. 
It is heartening to note that the use of experi- 
mental evaluation of curriculum design is recognized 
as a proper approach to improving the art (Warmbrod 
and Phipps, 1966, p*40), but based on the reported 
research this appears to be the exception rather 
than the rule. The implication is that even though 
nev;ly developed currlculums are often widely adopted, 
one simply isn't certain of their net educational 
and economic value. It is not surprising, therefore, 
that curriculum developers meet resistance to the 
adoption of their packages by teachers, principals 
and school superintendents. These parties are 
usually being asked to substitute a program or cur- 
riculum whose costs and results are familiar with 
an-.activity which clearly involves new costs but 
v;hose benefits are unknown or problematical at best. 
Thus, WG have yet to solve the continuing problem, 
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for instance, of the content and proper mix of 
general versus skill-specific training or educa- 
tion. This is a most serious issue but one which 
as yet has not been properly set' forth as a set 
of testable hypotheses much less analyzed in an 
orderly scicntifj.c fashion. 

Teacher PJduca t i on . . 
While sone aspects of teaching clearly are an art 

and are not subject to conscious policy manipulation, 

I 

there ^xist teaching techniques which can improve | 

the effectiveness of a given teacher. Regrettably . 

we must again report that while in general the 

methodology does exist to evaluate teaching tech- 

niques and, hence, establish their net impact, the 

use of such methods is an exception rather than the 

rule. The following exerpt is instructive (Phillips 

and Brigg, 1969, p. 22): 

Schnellc (1967) conducted a study to 
eva luate the affect iveness of an eight - 
week summer institute to train instru- 
mentation technology teachers. The 
institute v^as conducted at the State 
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University of New York Agricu].tural 
Technical College at Morrisville, Nev/ 
York, during the summer of 1966, The 
evaluation of the program was conducted 
by tv/o coirjnittces ; a Reg ional Vis it at ion 
Conimittee composed of four men knov/l- 
cdg cable in instrumentation and a Central 
Evaluation Coram.ittee com.posed of educators 
and industrialists , These committees 
concluded that the program v/as successful 
and out. standing , i^articularly in the 
areas of facilities, dedication of staff, 
and the technical content presented. Tv;o 
deficiencies v;erc noted: (a) the hetero- 
geneous background of the participants 
and (b) the lack of i^resentat ions on 
teaching techniques and methodology. 

Panels of experts clearly are needed to establish 

program ob j ect ives and specify the techniques one 

might adopt to reach the educational objectives of 

a given program. But, as v/ith any other social or 

educational program, the informed judgment of such 

experts, while possjb].e pr ima facie evidence of gross 

progi'am impact, is anything but conclusive evidence 

of net impact and net impact, to repeat the obvious, 

, is v:hat counts. 7\s with curriculum guides, then, 

the result is a plethora of general notions and 

techniques all competing for attention and use, but 
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with little to. distinguish among these in terms of 
their net quantitative impact- We submit that since 
such expert judgments have pretty v;ell distilled 
the main strategies of components of potentially 
successful teaching techniques, coherent experi- 
xnental designs be used to at least cull out the 
least effect ivo among these. Time and resources 
perm.itting, one can then, perhaps, attempt to measure 
net quantitative impact more precisely. To those 
v;ho would argue that the use of experimental design 
tampers with the destiny and human rights of in- 
dividuals, we would like to point out that what 
amounts to a continuing, yet casual "experimentation" 
also constitutes such tampering and while its lack 
of purposiveness may mcike it ideologically less 
objectionable as a method of establishing knowledge, 
its results (or rather, its lack thereof) may be more 
destructive of human destines and potential than more 
purposive experimentation simply due to the potential 
delay and lost opportunity or misguided information or 
advice- 

22 
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3 . Learning Processes > 

The big controversy . here is between conventional 
teaching methods and the several variants of self- 
instruction usually involving programmed queries and 
responses. Our survey of the surveys here report 
inconclusive/ or rather divergent results. In 
some cases techniques such as computer assisted 
learning or simulation exorcises are shown to be 
more effective than conventional techniques (Phillips 
and Brings, 19G9, p. 22 ff) while other studies have 
raiO\\ni no conclusive difference. 

The obvious conclusion emanat ing from this 
research effort was thi^.t the effectiveness 
of the inservicc self-instructional program 
could not be verified on the basis of 
criterion var iab3.e analysis . Tliis study / 
therefore, joins the ranks of the vast 
majority of investigations wh ich have 
attempted to assess the differential effec- 
tiveness of teaching methods and which have 
^ been able to report only '^nonsignif icant 

results." Greater confirmation is thus 
afforded the emerging tenet that it is 
extremely difficult to empirically demon- 
strate that one method of instruction is 
better than another. 
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It should be clear at the outset that, depending 
on the technique being offered, the skill or knowledge be 
ing imparted, snd the teacher and student characteristics 
involved, we will always find techniques that v/ork in 
, \B situation and not in another. This is what one 
T-7ould expect — that a given technique v;ill vary in its 
effectiveness given the other teaching inputs it is 
combined with. At present we have no careful summary 
and codification of the contexts in which, say, 
computer assisted learning works, and where it 
doesn't v;ork. We do not, therefore knov; if computer 
assisted learning .works (or fails) due to factors 
inherent in it or due to exogenous factors. Certainly, 
the confusion of these two possibilities is a clear 
and present danger to our ability to make informed 
judgments v^hen attempting to extend or generali^^ 
such techniques . 

A slightly different example of the problem we 
have at identifying the instructional components or 
characteristics that improve skill or educational 
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perfornvjnce is GXhibitGd by the analysis of skill 
acquisition in shorthand transcription by Joe M. 
Pullis (Price and Hopkins, 1970, pp. 54-55). Price 
and Hopkins report the findings of Pullis as follows: 

1. Shortliand dictation achievement is 
significantly relcited to the student's 
ability to construct accurate shorthc\nd 
outlines . 

2 . Shorthand transcription ability is s igni- 
f icantly related to the student * s ability 
to construct accurate shorthand outlines. 

3 . Shorthand dictat ion achievement is s ign i~ 

f icantly related to shorthanc]. transcription 
ability . 

4. Though shorthand transcription ability is 
significantly related to achievement in 
shorthand dictation, it is tlie student's 
abil ity to construct accurate shorthand 
outlines v;hich enables him to transcribe 
the outlines which he has written. 

Note iteras 1-3 v/herein achievement in a given operation 

is asserted to relate significantly to a closely related 

ability. The analysis is circular. The true deter-- 

minants of each skill are really what is at question. 

Thus, our knowledge gained from this study is zero 
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in terms of how to teach shorthand though we do get 
some predictive ability based on the apparent cor- 
relation. But/ fundamentally, these observations 
increase our knowledge of the learning of shorthand 
insignificantly. 

A final example in this entire area should 
suffice to illustrate how failure to adopt proper 
investigative techniques makes the ability to acquire 
meaningful information from demor.n crat ion and experi- 
mentation projects nugatory and hence, even if widely 
adopted, of unknov/n value. Gene R. Daugherty used 
a single group pre-test/post-test design to test a 
teaching plan and handbook for teaching beef marketing. 
There was, as indicated, no control group. The measure 
of outcome was not whether profits from beef market- 
ing increased but how high the subjects scored on an 
achievement test after the course. Thus, the effect 
of intervening variables on performance overtime was 
not accounted for and no theoretically convincing 
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IxYik was established between what was measured on 
the achievement test (even if the net change in test 
performance would have been large and significant) 
and any potential increase in profits from beef 
marketing. (Warmbrod and Phipps, 1966, pp. 56-57). 
Thus, even if this technique were widely adopted, 
one wouldn't know what its educat ional and social 
s ignif icancc was . 

Careful stucl'ies' using multivariate analytical 
techniques across a variety of situations can con- 
tribute to a clearer understanding of such issues. 
In particular, the use of a technique across a variety 
of situations, thus comprising a form of natural or 
quasi-experiment , with each situation comprising one 
observation, could provide the necessary data to 
sort out confusing but competing interpretations. 
In the meantime, we must affirm again that v;hat we 
have to go on is a useful first step in establishing 
our knowledge, but a step which must be built upon in 
order to clarify what is really happening. 

^ / 
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Capital Eaui pinent and 'Facilities . 

Adequate educational facilities clearly are 
important in the learning process. To cite one 
authoi'ity-, however, little definitive research 

has been done to establish criteria for effective 
facility planning." (Phillips and Briggs, 1969, p. 25) 

Elaborate planning guides for the design and con- 
struction of facilities as reported by Phillips and Briggs 
but the degree to v;hich these reflect the conclusions of 
actual experimentation or demonstration projects is 
unclear. (See also Warmbrod and Phipps, 1966, pp. 76-77) 

Support ive Servicos . 

Guidance . Guidance is hypothesized as being 
composed of three distinct elements: imparting 
occupat ional information ; appra isal of this informa- 
tion; and decisionmaking. The ability to assist 
students in making wise occupational choices is 
hampered by the fact that we have no clear under- 
standing of what vocational guidance tests measure, 
though their use is widespread. (Tuckman and Schaefer, 
1966, pp. ,'?7~28) 

28 



As of 1960, vocational guidance was not widely 
avciiliiblC/ being present in only about half of the 
Nation's high schools and in most cases the services 
were inadequate. (Phillips and Briggs, 1969, p. 14) 
However, this latter judgment is not necessarily 
based on solid evidence. A study by Robert O. Johnson 
attempted to test an interdisciplinary approach to 
vocational guidance which combined elements of English, 
social studies and vocational guidance. (Price and 
Hoi^k ins , 1970, pp. 66-67) Regrettably , the author 
had only 40 observations and no comparison group. 
Thus, the results of his considerable efforts carry 
no conclusive weight. In particular, v;hile everyone 
involved '*felt" this program was a "success," a simple 
Hawthorne effect could have accounted for any gross gains, 
assuming any objective existence of these at all. 
These types of demonstration projects are fundamentally 
untrustvjorthy as policy guides though they can often 
be a rich source of testable hypotheses. 
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A variety of work has been done to determine the 
personal, i:>Gycholog ica 1 and socio-demographic charac- 
teristics of vaxiovas types of vocational education 
students. Much of this descriptive v/ork is of 
potential interest for guidance counselors and could 
conce i.vably aid in proper matches between students 
and their potential long-run careers. But much of 
this work also appears to be of questionable value 
and a d i repass i onate observer trained to analyze 
programs or behavior based on hypothesized structural 
relationships among program inputs, student and 
teacher characteristics and program outputs would be 
hard pressed to discover a rationale for continuing 
such v7ork. Perhaps one illustration in this regard 
will suffice to establish the point. One analyst of 
distributive educati.on students found they visited 
the school health clinic more frequently than a com- 
parison group. \vhat operational hypothesis underlay 
i:he development of this datum? To what end was this 
datum collected? (Meyer and Logan, 1966, pp. 94 ff) 

• 
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Finally v;e cort^e to demonstrations and experiments 
designed to improve student's knowledge of labor market 
choices so thz\t they e n make more informed decisions 
about the labor market. These have a long history 
and have been popular. Many 'such experiments and 
demonstrations have be-en conducted by the Department 
of Health, Education and V7elfare. The Occupational 
Information System is a current demonstration of 
this type by the Department of Labor which builds 
on much of the HEW exi^erience. 

Problems with this kind of research are several, 
hov/ever» The basic one involves the establishment 
of a link between the deliverance of labor market 
information at one point in the working-education 
cycle of a young person and its effect on labor 
market perforraance once the person leaves school and 
enters the labor market. Much labor market informa- 
tion decays rapidly in its currency for a given market 
context so that information learned this year may not 
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be relevant six months later. Simple business cycle 
effects or structural shocks such as the current oil 
crisis can make information obsolescent within weeks. 
Thus, such demonstration projects shoul.d concentrate 
on determining the most effective way to deliver that 
information which is relevant to a given labor market 
at a given point in time. In general, this is not 
the focus of such projects, however. A meticulous 
investigation of the relative effectiveness of dif- 
ferent delivery systems (educational or informational 
product ion functions) are clearly not the rule . Thus ^ 
such studies ofton do little than demonstrate that it is 
socially, politically and administratively feasible to 
operate such programs. Their net impacts on ultimate 
occupational or labor market success are not known. 
Therefore, even though such prefects have been common, 
their value is not established though prima facie 
evidence would suggest that they would have some 
pos it ive impact. 
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Inno va tiv e Input Co mbinations . 

Within recent years several new concepts on how 
to recombine teaching inputs have developed. One's 
attention is first drawn to team teaching v;hich funda- 
mentally is an application of the principle of division 
of labor and specialization — a concept used extensively 
in post -secondary cducat ion where a graduate student 
receives a mix of interrelated and complementary 
instruction by a set of specialists in specific skill 
or technical areas. While it is very hard to unravel 
and identify the marg inal contr ibut ion of each teach- 
ing component on a student's performance due to the 
joint production involved, again, intuitively team 
teaching makes sense. It is so widely adopted that 
documentation of this fact is not of importance here. 
Vfhcxt is important is the phenomenon alluded to above-- 
joint effects on educational or job performance are 
difficult to identify and so we haven't any clear 
idea of the separate effects of team members on per- 
formance though the net impact of the team as a whole 
can be measured. 
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However, the most widely publicized innovation 
in input combinations has been Sidney Marland's 
"Career Education," The National Institute of Education 
is runn ing a var iety of exper iments here to test the 
effects of family -based, employer -based and other 
innovative career education experiments. 

The problems with these experiments have been 

0 

several, however. Most notable are these: sample 
sizes are too small; the experimental design integrity 
has been violated; no clear cut educational delivery 
system has been established in any of these experi- 
ments and the goals, and hence, delivery systems have 
constantly changed. As an example, the best experi- 
ment of the group, family-based career education, 
exhibits no regularity or uniformity in its educational 
delivery system or process. Efforts by Moayed-Dadkhah 
and Stromsdorfer to measure uniform relationships 
between program costs and intermediate educational 
outputs have met with failure. (Moayed-Dadkhad and 
Stromsdorfer, 1975) Apart from several data limitations 
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the biisic reason for this appears to be the diversity 
of educational treatments being offered as well as 
their changing structure over the life of the experi- 
ment. For instance, there are nov; 4,000 learning 
activity packages in 20 different educational or skill 
areas. There are less than 1^,000 observations avail- 
able to test for the net impact of these learning 
packages. Given the fact that random assignment pro- 
cedures have been violated, plus the rather small 
even if positive effects that may exist, there sijaply 
aren't enough observations to detect any measurable 
effects. The other career education experiments 
are in much worse shape. Hence, even if these 
projects are generalized and disseminated throughout 
the United States we have absolutely no idea of 
their ultimate educational, social or economic value 
over and above conventional vocational education 
approaches. Yet, Miller and Miller claim great 
things for career education. (Miller and Miller, 
1975, p. 55) 
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The Bottom Line 

Table 1 displays a wide variety of projects which 
have been funded under the Vocaticnal Educational Act 
and which in many cases have been widely disseminated 
acroF5s society. vrnat do we knov; of their ultimate educa- 
tional or social value? Perhaps the following quotes 
are ins^tructive since they appear to be the most current 
judgments based on the findings in the literature at 
th i s t: ixae : 

There wore times when .it almost seemed 
as though specific results v:ere not 
wanted, as if the results might not be 
v;hat the people who v;ere conducting the 
projects wished to believe. But part 
of the problem lay in not having definite 
information availabD.e at the beginning of 
a project relating to precisely v/hat ki.nds 
of data v;ere desired in the reports. 
Another deterrent has been the time' and 
cost involved in attempting to get evidence 
of impact. Only re^cently have valid in- 
striuTients, in which project directors can 
V • have confidence, begun to be developed. 

To the degree that either the l^ck of any 
valid impact measurement techniques or 
the possibility of negative results has 
hampered impact research in educational 
projects, there has been a loss of vital 
data to educational program management. 
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TADIiE 1 



h,^^ HF-llL i ^^SL^ Pi^Q_1_?'^^^L^ Fun clod under VEA^;, 
Parts C , D, and I 



ProjtTi. 
T5 L Ic 



r reject 
Cali f ornin 



Voria of Cor.si ruc- 
tion - V.'orld of 

Ohio 



First 
FunJ(Mj_ 

1903 



1965 



VI i:.;: Vital Infor- 
mer lion for educa- 
tion nnd Work 
Cal if ornia 



1965 



Ori^^inil 



Current 
Irip.ic t 



Nationwulc: survey 
resulted in chan^'.e 
of rc!quiro:r.cnts for 
instruction by FAA. 



CcM-plctc curriculum 
in each area inclu- 
dini; a3 1 r.ater ials, 
daily behavioral 
obiec lives, field 
tested in six 
States . 



Provided inforna- 
tion on nicroflclie 
about course offer- 
ings and jobs avail- 
able to students in 
San Die^o County, CA. 



150 certified avia- 
tion schools in 
U.S . , 18 other 
countries usin<^ 
revised curriculun, 
15,91.? graduates. 

Prop^rav^.fi taur,hi in 
all 50 Slates, 3 
provin.ccs of Canada 
America Dependent. 
School. s in Gcr:r!cnny, 
Franco, an.d Itoly. 
A20,000 students , 
2,200 school 

VlU-l projects in 3- 
Stales and Cuiir:, 
over 7,000,000 
students receiving;; 
information* 



37 



Table 1 - CouLlnucd 



rrojec t 
Til lo 



TERC: Technical 
Education Research 
Ccnter.s 
MassacliuGCCts 



First 
run clod 

1965 



Original 
Inpnc t 



T ^5 C : T < J h I : u 1 o ^ y 
for Children 
]lc\J Jersey 



1965 



Coufcreiice on 
In-.plor-oni: in^; Career 
Do vo 1 o p.T.e n t Th eery 
an J Hi^yoiirch 
throuf,!i Cnrr iculum' 
Vir;\iiiia 



1966 



CVIS : CorpuLerized 
Vocal; ional Infor- 
mal ion System 
Illinois 



1967 



Nationwide survey 
to dcterrilne avail- 
ability of biouedl- 
c a 1 cq u i pr.e at t e ch - 
nology p!-oj;rai.is . 
Developr.ont of 
CLirr iculun. 



Hands-on exp^^'uience 
and carciir a\;cirene3S 
for 572 children in 
one school district. 



Fifty part icipants' 
fror:i suveral States 
diij cussed career 
d Ci V e 1 V pip.c at as a 
p e r o p, a 1 g r o'.c t h 
process . This con- 
ference helped pro- 
vide the b.icl:.^;ro\ind 
for Sidney IlarJand's 
1971 statei'.ent sup- 
port j n c\ c; a r e o r d e v e 1- 
opment th.eory . 



Provided vocational 
guidance inf ornafion 
to students on a l-l- 
dialoj»uc vith com- 
puter basis . 



Current 
Inpac t 



Four najor tech- 
nical curricu la 
heinji tavij'Vat .in a 
total 'of 33 States, 
also providing tech- 
nical a ?; 3 i s t a nc c 
and curriculuiTi dev(*l- 
opmont in 20 other 
areas • 

50,000 children In 
AOo of the school 
districts in Ui:\: 
Jersey. Also in 
Nov; York. Workshops 
held in at least six 
oiiher States. 

"Career Education" 
beinjj pr«:sented to 
children in all 50 
Stales. In pilot 
year, 1 ess than 
1,5UO,00.1 cliildren 
K- 12 v;ero involved 
N'a t i.o nv id e . VJ i t h i n 
two years, over 
1,200,000 children 
in one State alone 
v;ere in pre-sccoa- 
dary [jjiidancc and 
pr e- vocational p r o- 
f^tains and in secon- 
dary Vocational 
Education. 

Over 10,000 student 
jL^sjns at original 
site. 75+ scliools 
adopting this pro- 
gram. Consortium 
nenbers in 38 States, 
and 2 otlier countries 



^ Originally funded by State of New Jersey and Ford Foundation because no 
provisions were nade iu the Vocational Education Act of 1963 for pre- 
vocat ional cler.ientary level prof-rans. Funded under the 1968 ^Vucndnents 
in many areas of Nev/ Jersey by 1970, « 
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Table 1 



- Coniinued 



Project 



First Ori.^.innl 



V-IXCS: Porfornancc objcc- 

Vorational-Tcch- tivos delcrrrJi^cd for 

nical llducolion occupationa] areas, 

Censor tlu:a of task analyiiis pro- 

cediirc developed, 

Ccorgla . 



Occupational Train- 1968 Provided nanpower 

in;; Infornation dcn^nnd and supply 

f;yrt.cn (OTlf,) InforrvUion to 

OhlMioi.u Vocational and 

Technical Educa- 
tion adr.tin j ntrators • 



Program F.c-virv for ; 
Jnitrovcrr.c-nt , Dcvc'l 
opr.'Cnt , end Lxp".*'^.- ; 
t;ion (ri'.il)i:) in 
Vocaliovial Lducation 
Onio 



1970 Initiated in (?]cvcn 

(1,1) of Oiiio's 10^'* 
Vocat i ona 1 r.duc.i- 
rion P3anning 
Dlntricls. 



SPAN: SvGtci.^s 
Prc{\rari Anprcach- 
in-^l Non-unc::>ploy- 
nent of Vocal i:*n.rJ 
Str.denti; 
Tcnnc::r>ee 



19 70 Job Guidance and 
Placement center 
including, co-op 
v'ork experience, 
clencntary Jnstvuc- 
tiona) television 
series for career 
• exploration. 



Current 
Inpa ct ^ 

Tark ar.alysis cata- 
lo[iS developed for 
over 200 jobs titles. 
In Florida alone, 
2A6 industrial and 
65 technical pro- 
grairs , Kxpandcd to 
include six other 
States, v;ith thVttc 
noro. expected to 
join this year. 

Information to 
several State 
a[',cnc J es concerned 
v:ith v-ianpov.'er 
planning',, f ollov;up 
of vocational 
students after one, 
three, and five 
years . Personnel 
t r a i n i r. info r n)3 liow, 

Cor.^plet ed pro2.ran 
reviev: for 90 plan- 
ninp, districts, vill 
have llie rest co:n- 
pl etc ^.M^tin.n two 
years • Unep.ploynent 
rate for vocational 
pro}p-ari ^;i-adi'a fes 0!:e- 
fourth the Nal ional 
average for that a&c^ C^'c- 

SPAN being inpler^ented 
ch r o u r.l 1 c u t Ke r.i i^l lis. 
Tennessee legisla- 
ture has passed a 
bill requiring, all 
hir.h schools to pro- 
vide Vocational Kdu- 
cation opportunities 
for at least 50% of 
the students State- 
wide, 
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Tabic 1 - Continued 



Project: 
Title 



YocaLional Follow- 
up System 
Minnesota 



First 
Fund ed 

1970 



Mountain Plains 

Froject 

Montana 



1971 



Forecasting; Guide: 
Fn «> 1 o >'r..en t / Fn r o J ,1 - 
mcnt; A Mnnagcn:ont 
Inf ornat ion System 
Project 
Wanhington 



1971 



Curriculum for 
PJanning (Project 
Next Step) 
Utah 



1971 



Orif;inal 
Imp act 



Data ijathering 
instrurrents developed 
to provide inforria- 
tion on studont popu- 
lation, pro^^ram ter- 
mination, and s tudcnt 
and employer follow- 
up. 



Twe ] ve f a :a i 1 :i e s i n 
pilot project. 
Total f ami J y career 
education inclu-Jinp^ 
home manat;r-ment and 
job training, f or 
d is ad van t a^jed-r^rj^" 
fci nilies . 

Devolopront of a 
UM'thcd of an.'ily::- 
ing and disp.layinj^ 
cnploynient and 
enrollment infor- 
ir.ation. 



Planning guide 
developed for local 
school districts . 
Combined with 5-* 
State project on 
curriculum planning 
funded throuf;h two 
other USOE agencies. 



Current 
Impact 



Follow-up data on 
students from all 
thirty-three (33) 
area vocational 
schools in Minnesota 
provided to State 
agencies and area 
school directors, 
used in program 
planning and eval- 
uation. 

Currently enrolling 
between 199 and 210 
families, 480 faiai-- 
lies have completed 
the program. 



lli^;hly reliable 

pi nnn ing in f o rr.iiit ion 

source u.sod by major 

State and local 

e d uc a t .1 on a 1 a <^ n c i e s 

throughout V'aohington, 

Directly influences 

State Plan. 0\'er 

100 occupations 

analyzed. 

Hundreds of work- 
shops conducted in 
niany States to train 
educators in methods 
of conducting needs 
assessments. State- 
wide commitment to 
career education in 
number of States, 
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Table 1 - ConLlnuod 



Trojcct 
• TJ t 1 e 



Toward Account- 
Ari::oiin 



Firm 
Fund o J 

197] 



IJiond lUitlcr- 

Ai>*ncy for Ini;tiuc- 
l ionnl *i ctj ('vir.i.on 



1972 



Orinlnnl 
Tr.ip acl 



Ucw };u{d;incc and 
counscli iig pro^r<nni 
piloted in 16 
scliools vith 8,200 
cLudcnts. 



Pc;jT>ounc'l from 3^^ 
Stiaos wrote por- 
tions of career 
dtjvolopiiiont pro- 
f:rnm - sor.c VEA 
funded , ir.os t m^; 
r>t:nl e funds . 



Current 



Systematically pi tinned 
and evaluated, 
object ivos- based 
instruct ional .Tnd 
co\]nf>elinf; activities. 
Methods for moasurlny 
outconiCTS in affeclive 
do:nain del ermi ncd . 
Fxpandi through- 
out district. 

Coi;ip] ete prej^ram 
nvai labl e being 
used lhro\ii;hout 
many States and in 
selected locat ions 
in others of the 3A ^ 



SOURCE: HJJ.LKR, KOHHRT and MILLHR, L7\RUE v; . (1974) Tiriv^i>qt;_of 
Vgcp tjj^n <*!_! Kci v\c a t^i on _ R c s o a .r c■l"L_^.■^_l; h e Fo c3 o r a 1 „p n_d 
S_teil.c? Lr.n;o,ls_, Project Bacollnc, Nortliorn Arizona 
State Un ivers ity , Flagstaff, Arizona . 
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Or, to further develop the situation we find our 
selves in, 

The tacit assumption seems to have been 
that the act of distributing money which 
is used to employ people to generate 
inf ormat ion will in\pact sor e educat ional 
system, will provide a service, will, 
through i.ts consumpt ion ; affect some 
aspect of the economic ^ see ial or pol i- 
tical system. Of course it willl But 
simplistic representations of impact 
should be strenuously avoided.... 

To report the impact of any intended 
activity requires more than a report on 
the activity or even the results of the 
activity. Before an activity is engaged 
in, the existing conditions intended to 
be changed by that activity must be 
determined. Somehow, the amount of 
resources required to effect the kind 
of change intended must be established, 
the duration of the activity must be 
approximated, and the general outcome 
of the impact projected- If the task 
attempted i too complex or expans ive 
for the lim3.ted resources available, 
then purposeful directions must be de- 
termined and implemented into policy 
statements before any activity is engaged 
in. And even though the establishment 
of clear criterion measures before the 
funding of Nationwide educational e-^forts 
may be politically hazardous, they must 
be made before the efforts reflected in 
those criteria can be measured. 
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In short, we do not lack for a richness, variety 
and ingenuity of ideas, Vocati.onal educators and educa- 
tors in general deserve considerable credit for their 
ingenuity in devising various curricula, techniques and 
educational strategies. They get low marks on being able 
to tell policymakers what the net impact o.f such activi- 
ties are, however* Thus, even tliough many of these 
innovations cire widely adopted, we can not be sure what 
the implications of these activities are for the students 
ijivolved or tho. nation as a whole. 
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IV, Concluding Remarks 

The ideal method to determine the actual impact 
of R6cD projects on vocational education is to design 
a questionnaire for Federal and State funding agencies 
in vocational education and educators and administra- 
tors in vocational education. This questionnaire should 
elicit: (1) information on the extent that these groups 
are familiar v/ith vocational education R&D findings; 
(2) the extent that they agree or do not agree with 
these (3) the extent that their policy decisions or 
instruction have been influenced by these findings, 
eitlier by modifying their curriculum content, instruc- 
tional devices, reorganization of educational programs, 
student services, or reallocation of funds v;ithin the 
educational system. Many of the available published 
research resu] ts only state their conclusions or recom- 
mendations. There is a time-lag for administers and 
other educators to implement these findings, if any. 
Even if some of the findings are adopted, there is no 
explicit or published record to show the actual con- 
tributions of R&D in vocational education. Because of 
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the limited time and resources available, this paper 
cannot pursue the mail questionnaire survey technique. 
Instead, this paper has evaluated the criteria for R€D 
projects that can be implemented and thus made inferences 
on the extent to v;hich these R&D projects may have af- 
fected vocational education. 

Based on the litejrature ava ilable , from a pos it ive 
point of viev;, the research and demonstration projects 
have provided general information on cost estimates and 
benefit-cost comparisons for administrators in planning 
and allocating funds for vocational education. Some of 
the demonstrat i.on projects have helped modify or improve 
instruction techniques. However, many limitations of 
these R&iD projects still exist which prevent their having 
a grociter contribution to improving vocational education. 
These limitations can be summarized as follows: (1) In 
general, many R&D projects do not have adequate research 
design or evaluation methodology; (2) descriptive research 
studies still abound in the literature although regression 
and correlation techniques have been increasingly applied 
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in the research; (3) Weakness in survey technique such 
as pr-obability sampling and meaningful questionnaire 
design are still evident; and (4) many studies are too 
v;eak to have durable impact either because of too small 
a sample size, too limited in terms of the population 
tested or the triviality of the problem tested in many 
demonstration projects . 

It should be noted that these stated weaknesses 
do not exist only in vocational lUxD projects (for 
example, sec Goldstein (1972) and Nay et al, (1973)). 
In fact, these v;eaknesses exist in many social and 
educational programs. Unlike R&D projects in the 
biological sciences, conceptually sophisticated social 
and educational R&D projects are a relatively recent 
occurence. R&D in vocational education is still in 
its infancy. There are many areas where as researchers 
and educators V7e can improve the current deficiencies 
in R&D projects so that they may have a greater actual 
contribution to vocational education. The statistical 
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and methodological technology now exists which can 
guarantee more reliable and general izable findings 
for RS<D in vocational education. The problem is that, 
for a variety of reasons , some of which are not valid , 
it is not alv.'ays used. It would seem to be an estab- 
lished fact, for instance, that control or comparison 
groups .v/ou Id be consistently employed, but we find 
this is not the case. Indeed, the principle is not 
even universally accepted. Yet, for such expensive 
research efforts to bear their promised fruit better 
techniques must indeed be adopted. In the meantime we 
are left with a v/ide variance in tlie quality and applica 
bility of R&D which can only serve to retard rather than 
facilitate the adoption of valid findings. 
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